Role-based tracking and surveillance

ABSTRACT

A method for surveilling a monitored environment includes classifying an individual detected in the monitored environment according to a role fulfilled by the individual within the monitored environment, generating a trajectory that illustrates movements and locations of the individual within the monitored environment, and detecting when the trajectory indicates an event that is inconsistent with an expected pattern for the role.

BACKGROUND OF THE INVENTION

The present invention relates generally to security and surveillance andrelates more specifically to methods for classifying and trackingsurveillance targets.

Many venues such as retail environments, factories, offices,transportation hubs, and the like employ surveillance systems in orderto ensure customer safety, national security, and/or operationalefficiency. Many conventional surveillance systems distribute large setsof surveillance devices (e.g., cameras, sensors, and the like) within amonitored area in order to detect illicit, unsafe, and/or unauthorizedevents. These devices generate surveillance data in the forms of streamsand/or data points that are typically forwarded to a central locationfor review (e.g., by a human operator).

The computational cost of processing all of the surveillance data,however, can be prohibitive. For instance, if the surveillance systemdeploys a plurality of devices, all of which are continuously generatingsurveillance data, a potentially enormous amount of surveillance datawill be generated. It may be difficult, if not impossible, for a humanoperator to efficiently review all of this data and therefore to respondin a timely manner to an event.

Moreover, events that may be normal for one type of person may beabnormal for another type of person. For instance, in a bankenvironment, it might be “normal” (i.e., likely not worthy of a securityalert) for a teller or cashier to venture behind the counter, but“abnormal” (i.e., potentially worthy of a security alert) for a customerto do the same. Many conventional surveillance systems, however, havedifficulty distinguishing between normal and abnormal events based onthe type (or role) of the person involved in the events.

SUMMARY OF THE INVENTION

A method for surveilling a monitored environment includes classifying anindividual detected in the monitored environment according to a rolefulfilled by the individual within the monitored environment, generatinga trajectory that illustrates movements and locations of the individualwithin the monitored environment, and detecting when the trajectoryindicates an event that is inconsistent with an expected pattern for therole.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited features of the presentinvention can be understood in detail, a more particular description ofthe invention may be had by reference to embodiments, some of which areillustrated in the appended drawings. It is to be noted, however, thatthe appended drawings illustrate only typical embodiments of thisinvention and are therefore not to be considered limiting of its scope,for the invention may admit to other equally effective embodiments.

FIG. 1 is a block diagram illustrating one embodiment of a system forrole-based tracking and surveillance, according to the presentinvention;

FIG. 2 is a flow diagram illustrating one embodiment of a method forperforming role-based tracking and surveillance, according to thepresent invention; and

FIG. 3 is a high-level block diagram of the role-based tracking andsurveillance method that is implemented using a general purposecomputing device.

DETAILED DESCRIPTION

In one embodiment, the invention is a method and apparatus forrole-based tracking and surveillance. Embodiments of the inventionautomatically detect abnormal behavior of specific individual types orroles based on surveillance data generated by a plurality ofsurveillance devices. Within the context of the present invention, a“role” refers to the social and/or professional capacity (including thebehaviors, rights, and obligations associated with that capacity) withinwhich an individual finds him or herself in a monitored environment. Forinstance, in a retail environment, an individual may fulfill the role of“customer,” “cashier,” “manager,” “security personnel,” or the like.Abnormal events can be automatically detected by comparing a detectedindividual's observed actions with historical patterns (e.g.,predetermined “normal” or allowed conduct and activities) for the roleaccording to which the individual is classified. The disclosed systemsupports even large-scale surveillance systems because it does notrequire or rely on the identification of the detected individual or onpre-defined behavior types. Although the invention is described withinthe exemplary context of human tracking and surveillance, the sametechniques described herein may be used to track and monitor non-humansurveillance targets (e.g., animals, vehicles, etc.). Moreover, althoughthe invention is described within the exemplary context of securityapplications, the disclosed techniques could also be implemented todetect non-security related events.

FIG. 1 is a block diagram illustrating one embodiment of a system 100for role-based tracking and surveillance, according to the presentinvention. In one embodiment, the system 100 includes two maincomponents: plurality of sensors 102 ₁-102 _(n) (hereinaftercollectively referred to as “sensors 102”) and a behavior assessor 104.The sensors 102 and the behavior assessor 104 cooperate to automaticallyclassify individuals according to role and to detect behaviors thatdeviate from behavior expected of the roles.

The sensors 102 provide surveillance data for processing by variouscomponents of the behavior assessor 104. These sensors 102 may includeone or more of: imaging sensors (e.g., still cameras, video cameras,thermographic cameras, or the like), biometric sensors (e.g.,fingerprint sensors, ocular sensors, voice sensors, or the like), orelectronic scanners (e.g., for scanning machine-readable tags or labels,such as radio frequency identification tags, bar codes, or the like).The sensors 102 collect surveillance data from various physicallocations within a monitored environment. For instance, any one or moreof the sensors 102 may be positioned to collect surveillance data at theentrances and exits of the monitored environment, from specificlocations within the monitored environment (e.g., restricted access orhigh-traffic areas), or from any other location.

The behavior assessor 104 receives surveillance data from the sensors102 and generates an alert when analysis of the surveillance dataindicates the occurrence of an abnormal event. To this end, the behaviorassessor 104 comprises at least: a plurality of feature detectors 106₁-106 _(n) (hereinafter collectively referred to as “feature detectors106”), a feature fusion engine 108, a global classifier 110, a globaltracker 112, a pattern matcher 114, and a pattern builder 116. Any ofthe behavior assessor components 106-116 may comprise a processorconfigured to perform specific functions related to role-based trackingand surveillance. In addition, the behavior assessor 104 is incommunication with a pattern database 118. Although the pattern database118 illustrated in FIG. 1 is depicted as remote storage (i.e., separatefrom the behavior assessor 104), in alternative embodiments, the patterndatabase may be integrated with the behavior assessor 104 as localstorage.

The feature detectors 106 receive raw surveillance data from the sensors102 and extract features from the surveillance data. The featurescomprise characteristics depicted in the surveillance data that may aidin identifying the role of and/or tracking an individual present in themonitored environment. For instance, a feature detector 106 thatreceives thermographic images from a thermographic camera may extract athermal marker or pattern from the thermographic images that indicates aparticular role. Although FIG. 1 illustrates a one-to-one correspondencebetween sensors 102 and feature detectors 106, the system 100 is not solimited. For instance, a given feature detector 106 might receivesurveillance data from a plurality of sensors 102 of the same type (e.g.a plurality of video cameras). Alternatively, a given sensor 102 mayprovide surveillance data to a plurality of feature detectors 106 thateach implement different algorithms for feature extraction.

The feature fusion engine 108 receives the features extracted by thefeature detectors 106 and correlates the features into groups thatpertain to common individuals or roles. For instance, the feature fusionengine 108 may determine that a thermographic marker extracted by afirst feature detector 106, a facial image extracted by a second featuredetector 106, and a radio frequency identification tag extracted by athird feature detector 106 all depict the same person or role. Having aplurality of extracted features related to a given individual mayincrease the probability of correctly identifying the individual's roleand improve the ability of the system 100 to track the individualthroughout the monitored environment.

The global classifier 110 receives the groups of features generated bythe feature fusion engine 108 and assigns a class to each of the groupsof features. In one embodiment, the class assigned to a group offeatures identifies a characteristic of the individual to whom the groupof features pertains. In one particular embodiment, the globalclassifier 110 classifies groups of features based on the roles of theassociated individuals in the monitored environment. For example,individuals in a retail environment may fulfill a set of roles including“customer,” “cashier,” “manager,” “security personnel,” or the like;thus, the groups of features extracted from the retail environment maybe classified according to these roles. The global classifier 110identifies the role/class that is best implied by a given group offeatures.

The global tracker 112 receives the classified groups of features anduses them to track the associated individuals or roles throughout themonitored environment. Thus, the global tracker 112 associates atrajectory with each group of features that records the movements andlocations of the associated individual or role within the monitoredenvironment.

The pattern matcher 114 receives the trajectories and associated classesfrom the global tracker 112 and uses this information to determine whenan event has occurred that represents a potential abnormality (e.g., abreach of security). In one embodiment, the pattern matcher 114 detectssuch events by comparing a given trajectory to an historical behaviorpattern for the class associated with the trajectory. For instance, ifthe pattern matcher 114 receives a trajectory associated with the classof “customer,” the pattern matcher 114 would compare the trajectory toan historical behavior pattern for the class of “customer.” In oneembodiment, the historical behavior patterns for various classes arestored in the pattern database 118.

The pattern builder 116 also receives the trajectories and associatedclasses from the global tracker 112, but uses this information to learnand build behavior patterns associated with the various classesindicated by the groups of features. The learned patterns are stored inthe pattern database 118 for use by the pattern matcher 114 as describedabove.

Although the system 100 is illustrated as comprising a plurality ofindividual components that perform discrete functions, it will beappreciated that any two or more of the illustrated components may becombined in a single component that performs multiple functions.Additionally, although the system 100 is illustrated as a containedsystem, it will be appreciated that the various components of the system100 may be physically distributed throughout the monitored environment(although still contained within the physical boundaries of themonitored environment), and some of the components may even be locatedoff-site (i.e., outside the physical boundaries of the monitoredenvironment). To this end, the various components of the system 100 mayinclude a combination of wireless and physically connected devices.

FIG. 2 is a flow diagram illustrating one embodiment of a method 200 forperforming role-based tracking and surveillance, according to thepresent invention. The method 200 may be performed, for example, by thesystem 100 illustrated in FIG. 1. As such, reference is made in thediscussion of the method 200 to various elements depicted in FIG. 1.However, it will be appreciated that the method 200 may also beperformed by systems having alternate configurations.

The method 200 begins at step 202 and proceeds to step 204, where thesensors 102 collect surveillance data from a monitored environment. Forinstance, the surveillance data may include substantially real-time datacollected by one or more of the sensors 102 that allows the system 100to uniquely identify individuals within the monitored environment.Within the context of the present invention, the ability to “uniquelyidentify” and individual does not necessarily imply that theindividual's identity (e.g., name) is recognized or revealed. Rather,embodiments of the present invention associate sets of features with acommon individual as a means of recognizing the individual (anddistinguishing the individual from others) when he or she moves todifferent locations within a monitored environment. As discussed above,the surveillance data may include images, biometric data, electronicdata, or other types of information collected from the monitoredenvironment. For instance, the images may include still and/or videoimages, which would allow the system 100 to identify and trackindividuals by their appearances. The biometric data may includefingerprints, ocular features, gait, or the like, which would allow thesystem 100 to identify and track individuals by their individualfeatures. The electronic data may include bar codes, radio frequencyidentification tags, or the like, which would allow the system 100 toidentify and track individuals by electronic and/or electromagneticsignals.

In step 206, the feature detectors 106 detect and extract features fromthe surveillance data collected in step 204. In one embodiment, theextracted features include information that helps to identify the rolesof individuals present in the monitored environment distinguish amongindividuals present in the monitored environment. For instance, featuresextracted from a still image of an individual might include a uniform ora badge that indicates the individual's role. Features extracted from aradio frequency identification tag attached to an individual mightinclude an access authorization or license. Features extracted from athermographic image might include thermal markers (e.g., patterns thatmay be recognized by a pattern recognition algorithm) embedded in anindividual's clothing. Because thermal markers are substantiallyinvisible to the human eye, they may be especially useful for markingundercover security personnel, who may need to access restrictedlocations without explicitly broadcasting their roles. Thermal markersare also less sensitive to environmental conditions that would otherwiseaffect tracking using conventional visual markers (e.g., low-lightconditions).

In step 208, the feature fusion engine 108 correlates the extractedfeatures into groups. A group of features created by the feature fusionengine 108 includes features that are believed to identify a commonindividual (e.g., a thermographic marker and a facial image associatedwith the same person).

In step 210, the global classifier 110 attempts to classify the groupsof features. In one particular embodiment, the global classifier 110attempts to recognize the role associated with each group of features.For example, in a retail environment, a given group of features mightindicate that the individual from whom the features come is a customer,a cashier, a manager, a security guard, or the like. In a sport eventenvironment, a given group of features might indicate that theindividual is a player, a coach, a spectator, a referee/umpire, asecurity guard, a vendor, or the like. In some cases, the globalclassifier 110 may not be able to classify a given group of features. Insuch an event, the global classifier 110 may wait to receive additionalinformation or features from the feature fusion engine 108, in case theadditional information helps to resolve the classification.

In step 212, the pattern matcher 114 matches the classes to historicalbehavior patterns associated with the classes. The historical behaviorpattern for a given class represents movements and activities that areconsidered “normal” (i.e., likely not worthy of an alert) for the roleassociated with the given class. In one embodiment, the historicalbehavior patterns are retrieved from the pattern database 118.

In step 214, the global tracker 112 tracks the movements of theindividuals associated with the groups of features throughout themonitored environment. This creates a series of trajectories thatillustrate the individuals' real-time movements and behaviors in themonitored environment. In one embodiment, the individuals are trackedonly for as long as they remain within the monitored environment; oncean individual exits the monitored environment (which may be indicated bythe global tracker's inability to detect the individual's presence), theglobal tracker 112 ceases to create a trajectory for the individual.

In step 216, the pattern matcher 114 compares the trajectories to thehistorical behavior patterns for the corresponding classes. Inparticular, the pattern matcher 114 observes the trajectories in orderto confirm that the trajectories are substantially consistent with theexpected historical behavior patterns to which they are matched.

In step 218, the pattern matcher 114 determines whether any of thetrajectories have deviated from the historical behavior patterns towhich they are matched. In one embodiment, deviations are detected bycomputing a measure of similarity or dissimilarity between thetrajectories and the associated historical behavior patterns. In oneembodiment, the pattern matcher 114 is tolerant to a thresholddeviation; however, any deviation beyond the threshold is consideredabnormal.

If the pattern matcher 114 concludes in step 218 that a trajectory hasdeviated from an historical behavior pattern to which it is matched,then the pattern matcher 114 issues an alert in step 220. In oneembodiment, issuing the alert includes sending a message includingdetails of the deviation (e.g., the class of the individual associatedwith the deviation, the action that deviated from the historicalbehavior pattern, the location at which the deviation occurred, etc.) toa central control location for review. In another or further embodiment,issuing the alert includes activating an alarm (e.g., an audible and/orvisible alarm) or taking other cautionary or corrective measures (e.g.,activating locks in restricted areas). Once an alert has been issued,the method 200 returns to step 214. The global tracker 112 and thepattern matcher 114 continue to track individuals and to compare theirmovements to historical behavior patterns until the individuals exit themonitored environment.

If the pattern matcher 114 concludes in step 218 that no trajectory hasdeviated from the historical behavior pattern to which it is matched,then the method 200 returns to step 214. The global tracker 112 and thepattern matcher 114 continue to track individuals and to compare theirmovements to historical behavior patterns until the individuals exit themonitored environment.

As discussed above, role-based tracking and surveillance according tothe present invention relies in part on a database of historicalbehavior patterns for the various roles with which an individual in amonitored environment may be associated (e.g., pattern database 118).Although behavior patterns may be pre-programmed into the system 100,the system 100 is also capable of learning new roles and associatedbehavior patterns. In addition, the system 100 may learn new featuresand/or behavior patterns associated with previously established roles.Ongoing learning allows the system 100 to perform more accurate trackingand surveillance by remaining up-to-date and by building a customizedknowledge base for the monitored environment in which the system 100 isemployed.

Although the present invention is largely described within the contextof the access control, it is noted that the methods disclosed herein maybe advantageously deployed in other contexts. For instance, otherembodiments, the present invention may be used to determine whetheractions taken by a particular individual are consistent with his or herrole. As a specific example, embodiments of the present invention may beemployed to ensure security and adherence to safety protocols bymonitoring employees in a workplace (e.g., a factory or laboratory).

In further embodiments, the ability to detect different behaviorpatterns may support the provision of differentiated or customizedservices. For instance, different sub-types of customers may be detectedin a retail environment, and customized goods or services could beoffered to the customers based on sub-type.

FIG. 3 is a high-level block diagram of the role-based tracking andsurveillance method that is implemented using a general purposecomputing device 300. In one embodiment, a general purpose computingdevice 300 comprises a processor 302, a memory 304, a tracking andsurveillance module 305 and various input/output (I/O) devices 306 suchas a display, a keyboard, a mouse, a stylus, a wireless network accesscard, an Ethernet interface, and the like. In one embodiment, at leastone I/O device is a storage device (e.g., a disk drive, an optical diskdrive, a floppy disk drive). It should be understood that the trackingand surveillance module 305 can be implemented as a physical device orsubsystem that is coupled to a processor through a communicationchannel.

Alternatively, the tracking and surveillance module 305 can berepresented by one or more software applications (or even a combinationof software and hardware, e.g., using Application Specific IntegratedCircuits (ASIC)), where the software is loaded from a storage medium(e.g., I/O devices 306) and operated by the processor 302 in the memory304 of the general purpose computing device 300. Thus, in oneembodiment, the tracking and surveillance module 305 for role-basedtracking and surveillance, as described herein with reference to thepreceding figures, can be stored on a computer readable storage mediumor device (i.e., a tangible or physical article such as RAM, a magneticor optical drive or diskette, and the like, rather than a propagatingsignal).

It should be noted that although not explicitly specified, one or moresteps of the methods described herein may include a storing, displayingand/or outputting step as required for a particular application. Inother words, any data, records, fields, and/or intermediate resultsdiscussed in the methods can be stored, displayed, and/or outputted toanother device as required for a particular application. Furthermore,steps or blocks in the accompanying figures that recite a determiningoperation or involve a decision, do not necessarily require that bothbranches of the determining operation be practiced. In other words, oneof the branches of the determining operation can be deemed as anoptional step.

While the foregoing is directed to embodiments of the present invention,other and further embodiments of the invention may be devised withoutdeparting from the basic scope thereof. Various embodiments presentedherein, or portions thereof, may be combined to create furtherembodiments. Furthermore, terms such as top, side, bottom, front, back,and the like are relative or positional terms and are used with respectto the exemplary embodiments illustrated in the figures, and as suchthese terms may be interchangeable.

What is claimed is:
 1. A computer readable storage device containing anexecutable program for surveilling an environment monitored by a sensor,where the program causes a processor to perform steps comprising:extracting data about an individual present in the environment from anelectronic signal provided by the sensor; classifying the individualaccording to a role fulfilled by the individual within the environmentwithout recognizing an identity of the individual, using the dataextracted from the electronic signal; generating a trajectory thatillustrates movements and locations of the individual within theenvironment; and detecting, subsequent to the classifying, when thetrajectory indicates an event that is inconsistent with an expectedpattern of behavior for the role.
 2. The computer readable storagedevice of claim 1, further comprising: activating a tangible responsivemeasure in response to the detecting.
 3. The computer readable storagedevice of claim 1, wherein the classifying comprises: correlating thedata extracted from the electronic signal with surveillance datacollected from another sensor to produce a group of features associatedwith the individual; and identifying the role as implied by the group offeatures.
 4. The computer readable storage device of claim 3, whereinthe another sensor is positioned to collect the surveillance data fromthe environment.
 5. The computer readable storage device of claim 4,wherein the surveillance data includes an image of the individual. 6.The computer readable storage device of claim 5, wherein the image is athermographic image.
 7. The computer readable storage device of claim 6,wherein the thermographic image depicts a thermal marker.
 8. Thecomputer readable storage device of claim 4, wherein the surveillancedata includes biometric data.
 9. The computer readable storage device ofclaim 1, wherein the data extracted from the electronic signal includesmachine-readable data.
 10. The computer readable storage device of claim1, wherein the expected pattern of behavior is retrieved from a databasestoring a plurality of behavior patterns associated with various roleswithin the environment.
 11. The computer readable storage device ofclaim 1, wherein the expected pattern of behavior indicates historicalbehavior associated with the role.
 12. The computer readable storagedevice of claim 1, wherein the detecting comprises: computing a measureof similarity between the trajectory and the expected pattern ofbehavior, wherein the event deviates from the measure of similaritybeyond a threshold.
 13. The computer readable storage device of claim 1,wherein the detecting comprises: computing a measure of dissimilaritybetween the trajectory and the expected pattern of behavior, wherein theevent deviates from the measure of similarity beyond a threshold. 14.The computer readable storage device of claim 1, wherein the tangibleresponsive measure comprises an activation of a lock in a restrictedarea of the environment.
 15. The computer readable storage device ofclaim 14, wherein the tangible responsive measure comprises a messagesent to a central control location.
 16. The computer readable storagedevice of claim 15, wherein the message includes detailed datapertaining to the event.
 17. The computer readable storage device ofclaim 14, wherein the tangible responsive measure comprises anactivation of an alarm.
 18. The computer readable storage device ofclaim 14, wherein the tangible responsive measure comprises animplementation of a cautionary measure.
 19. A surveillance system,comprising: a plurality of sensors positioned to collect surveillancedata from a monitored environment, wherein the plurality of sensorsincludes a sensor to detect an electronic signal; a feature extractor toextract features from the surveillance data, wherein the featureextractor extracts data about an individual present in the monitoredlocation from the electronic signal; a classifier to classify theindividual according to a role fulfilled by the individual within themonitored environment; without recognizing an identity of theindividual, using the data extracted from the electronic signal; atracker to generate a trajectory that illustrates movements andlocations of the individual within the monitored environment; and apattern matcher to detect, subsequent to the classifier classifying theindividual, when the trajectory indicates an event that is inconsistentwith an expected pattern of behavior for the role.
 20. The surveillancesystem of claim 19, wherein the plurality of sensors comprises at leastone of: an imaging sensor, a biometric sensor, or an electronic scanner.